Method and unit for generating high dynamic range image and video frame

ABSTRACT

An embodiment of the invention relates to a method for High Dynamic Range (HDR) image creation by generating a mean difference curve for use in aligning the images of a sequence of images taken with different exposures, wherein the difference in exposure might derive from a difference in exposure time or a difference in aperture size. A further embodiment of the invention relates to a method for HDR image creation by generating images with different exposures from a single image. A further embodiment of the invention relates to a method for HDR video creation by generating frames with different exposures from a single frame.

An embodiment of the invention relates to a method for High DynamicRange (HDR) image creation by generating a mean difference curve for usein aligning the images of a sequence of images taken with differentexposures. A further embodiment of the invention relates to a method forHDR image creation by generating images with different exposures from asingle image. A further embodiment of the invention relates to a methodfor HDR video creation by generating frames with different exposuresfrom a single frame.

BACKGROUND

In HDR image creation, multiple images are taken at different exposuresand then blended in order to have an image with a dynamic range higherthan the dynamic range of a single image. Within this blending procedurethose images with smaller exposures, which contain more information inbright areas, improve the dynamic range within these bright areas of theimage, and other images with higher exposure improve the dynamic rangewithin dark areas. Therefore by blending these images of differentexposure the dynamic range of the resulting image can be increased.

However, an image sequence taken with a handheld camera contains globaland local motion, wherein global motion derives e.g. from movement ofthe camera and local motion derives e.g. from movement of objects to beimaged. Thus, for HDR image creation those images should be aligned witheach other before blending them. Hereinafter the term luminance is usedequivalently to the term radiance.

It is a problem that state of the art methods for motion estimationcannot align images with different exposures efficiently, because thereis no linear change in luminance between images taken with differentexposures due to non-linear post processing inside the camera after animage is captured.

Conventional motion estimation does not work with non-linear luminancechange, since state of the art techniques crop the images, which resultsin the loss of information and in artifacts, which appear in the alignedimages due to motion when applying conventional techniques. Therefore,when using state of the art techniques it is not possible to preservethe dynamic range of an image in all image regions.

In conventional block based motion estimation, a block in the currentimage is matched with the neighboring blocks in a reference image andthe block which has for example least mean absolute difference (MAD) istaken as a predicted block.

Also a method of fading images, in which a global DC of each image issubtracted from each pixel value in calculating the mean absolutedifference does not work for images having different exposure becausethe change in exposure is luminance dependent. Therefore both theconventional method and the method for fading images do not work for theimages having different exposures because the change in exposure isluminance dependent.

It has also not been possible so far to generate different exposureimages from a single image or to generate different exposure frames froma single frame in HDR video.

Therefore it is an object of the invention to provide a method and unitfor improving the generation of high dynamic range image based on areference among different exposure images so that aligned images ofdifferent exposure are created for use in HDR imaging.

BRIEF SUMMARY

The problem is solved by a method according to claim 1, a unit accordingto claim 8 and a computer readable medium according to claim 12. Furtherembodiments are defined in the dependent claims.

Further details of the invention will become apparent from aconsideration of the drawings and ensuing description.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of embodiments and are incorporated in and constitute apart of this specification. The drawings illustrate embodiments andtogether with the description serve to explain principles ofembodiments. Other embodiments and many of the intended advantages ofembodiments will be readily appreciated as they become better understoodby reference to the following detailed description. The elements of thedrawings are not necessarily to scale relative to each other. Likereference numerals designate corresponding similar parts.

FIG. 1 is a block diagram illustrating the basic intention of findingthe motion vector.

FIG. 2 is a block diagram illustrating a method for generating a HDRimage.

FIG. 3 a is a block diagram illustrating the procedure of taking imagesat different exposures with non-linear changes of the pixel value. FIG.3 b shows, how a pixel value (in the example the considered pixel valueis “10”) is varied for different exposures. This variation/change in thepixel value can also be considered as a transition between the referenceimage and one of the n−1 further images with exposure values eV1, eV2,etc.

FIGS. 4 a and 4 b show arrays of reference values for a transition fromreference image with exposure value eV0 to a further image with exposurevalue eV1 and for a transition to another further image with exposurevalue ev2.

FIGS. 5 a and 5 b correspond to the transitions shown in FIGS. 4 a and 4b and show the “mean difference value” for each pixel value of thetransition of FIG. 4 a or 4 b.

FIG. 6 shows the mean difference values of all transitions for eachpixel value and is the final look-up table to be stored.

Different uses of the look-up table are illustrated in FIGS. 7 and 8.

FIG. 9 shows a generation unit for generating a high dynamic rangeimage.

DETAILED DESCRIPTION

In the following, embodiments of the invention are described. It isimportant to note, that all described embodiments in the following maybe combined in any way, i.e. there is no limitation that certaindescribed embodiments may not be combined with others. Further, itshould be noted that same reference signs throughout the figures denotesame or similar elements.

It is to be understood that other embodiments may be utilized andstructural or logical changes may be made without departing from thescope of the invention. The following detailed description, therefore,is not to be taken in a limiting sense, and the scope of the presentinvention is defined by the appended claims.

It is also to be understood that the features of the various embodimentsdescribed herein may be combined with each other, unless specificallynoted otherwise.

FIG. 1 is a block diagram illustrating the basic intention of findingthe motion vector. In an image 10 a block 11 is located having pixels12, wherein each pixel has a pixel value. In another image 20 with adifferent exposure the pixel values of block 11 will correspond to pixelvalues of pixels 22 of a block 21 with those pixel values of block 21being different to the pixel values of block 11 due to the differentexposure between images 10 and 20. If however there is motion betweenimage 20 and image 10, this change in pixel values correlates also witha movement of the corresponding blocks within the plane of the image andcan only be determined if the motion vector 11-21 between image 10 andimage 20 is known.

FIG. 2 is a block diagram illustrating a method for generating a meandifference curve. In step S200 a sequence of n images is taken with acamera without local or global motion in order to generate an ideal“reference case” and can be realized with the camera mounted on atripod. One of these images is chosen as a reference image and theothers serve as auxiliary or further images, wherein an exposure valuefor the reference image is named eV0 and for the further images eV1,eV−1, eV2, etc. Then it is noted, how a certain pixel value, which mightappear several times within the reference image, changes betweenreference image and a further image by a difference value, which is notnecessarily constant for all pixels of the reference image. In step S210for each pixel value of the reference image a plurality of differencevalues for a further image is determined and grouped as an array. Instep S220 for each of the n−1 references (transitions) between thereference image and the n−1 further images a difference curve iscalculated. This difference curve represents a graph which indicates amean difference value for each pixel value. This mean difference valueis calculated as a mean value from the difference values of the array ofthe pixel value. In a step 230 the mean difference values are stored asthese mean difference curves in a look-up table, which is then used forthe camera with which the values have been calculated.

As shown in FIG. 3 a a sequence of n images is taken with differentexposure values eVx, wherein the difference in exposure might derivefrom a difference in exposure time or a difference in aperture size.This has to happen without local or global motion in order to generatean ideal “reference case”. Therefore any motion during taking the imagesshould be avoided, e.g. by taking the images with a tripod. One of thesen images is determined to be used as a reference image while the othern−1 images are used as further images for calculating differences intheir pixel values relative to the pixel values of the reference image.Thereby the term pixel value comprises information on the radiance.Pixel values might be for example 0 to 255.

In an embodiment of the method an image could be chosen as the referenceimage, which is not the image with the highest or the lowest exposure,in order to use the effect of improving the range both for dark and forbright areas with similar efficiency. Afterwards a difference isconsidered for each pixel value within the reference image, wherein thedifference values from reference image to the n−1 further images aredetermined.

FIG. 3 b shows how a pixel value (in the example the considered pixelvalue is “10”) is varied during transitions from the reference image tothe further images with exposure values eV1, etc.

So for each pixel value of the reference image an array of differencevalues is calculated for n−1 transitions (from the reference image ton−1 further images). For example in FIG. 3 b the transition from eV0 toeV1 of pixel value 10 results in pixel values 14, 13 and 15. Theresulting array of difference values for pixel value 10 is shown in FIG.4 a for transition to eV1 and in FIG. 4 b for transition to eV2. Thiscalculation is possible in a direct comparison of the pixels, sincethere is no movement within the sequence of images. The calculation isdone for all n−1 transitions from reference image to the n−1 furtherimages. Afterwards each array is “shrinked” to one value being the meandifference value of the corresponding pixel value.

While FIGS. 4 a and 4 b show arrays only for pixel value 10, thecorresponding FIGS. 5 a and 5 b take for the array at pixel value 10 andfor each further array of other pixel values one mean value, so that acurve is generated, which indicates one mean difference value for eachpixel value. This curve shown in FIGS. 5 a and 5 b for transitions fromthe reference image to the further image with exposure value eV1 and tothe further image with exposure value eV2 is also called mean differencecurve.

FIG. 6 shows a look-up table LUT with all mean difference values for then−1 transitions (on the y-axis) for pixel values 0 to 255 to be filledin. For reasons of easier illustrations there are only two meandifference values for two different transitions illustrated for pixelvalue 10. If such look-up table were filled in continuously it couldalso be shown as a three-dimensional graph spanned above the plane ofthe different transitions (y-axis of FIG. 6) over the pixel values(x-axis of FIG. 6). This look-up table is then stored and it is possibleto store a plurality of look-up tables for a plurality of differentfocal lengths and/or different ISO sensitivities.

A higher number n of different exposure images tends to provide a betterweighting.

When HDR images are taken with a handheld camera and there is motionwithin the image sequence, alignment of the images is necessary beforemerging them to create an HDR image. If the images, with local or globalmotion, were not aligned with each other, there would be artifacts inthe created HDR image.

As already explained above the look-up table is generated once for animage taking unit, e.g. a photo camera. Once being generated the look-uptable can be used for aligning a sequence of images taken by the samecamera even including motion.

So, once knowing the luminance-exposure-dependency represented by thelook-up table, the alignment of a sequence of images for creation of anHDR image can be done without artifacts, even with motion within thesequence of different exposure images, and does not necessarily usefurther manual tuning of any parameters.

As already indicated, different look-up tables for different focallengths and for different ISO sensitivities can be stored for onecamera.

One use of the look-up table is illustrated in FIG. 7 showing a methodfor creating a high dynamic range image according to a furtherembodiment of the invention. The sequence of images taken for creating aHDR image is aligned by use of the look-up table and after linearizationof the aligned images the images can be blended to result in one HDRimage without causing artifacts.

In FIG. 7 the look-up table is already known by taking the firstsequence of images without motion as described above and shown in FIG.2. Then in step S710 with the same camera a sequence of images is taken.In step S720 this taken sequence is aligned based on the look-up tablealready known for this camera. A further step S730 of linearizing thealigned images may follow. In step S740 the images are blended in orderto generate a HDR image.

This method might also be applied to video frames.

Furthermore, as shown in FIG. 8, the look-up table can also be used forsome kind of “artificial HDR” image creation by generating differentexposure images from a single image with no further need for alignmentbefore blending. Blending of this artificially generated sequence willenhance the contrast of the original single image. In an analogousmanner also HDR video can be created artificially by applying the methodto a single frame thereby generating artificial different exposureframes for blending which will enhance the contrast of the originalsingle frame.

Accordingly also in FIG. 8 the look-up table is already known by takingthe first sequence of images without motion as described above and shownin FIG. 2. Then in step S810 with the same camera a single image istaken. In step S820 from this taken single image a plurality ofartificial further images is generated on the basis of the look-up tablealready known for this camera. In step S830 the taken single image andthe artificially created further images are blended in order togenerated a HDR image.

This method might also be applied to video frames.

FIG. 9 shows a generation unit 100 for generating a high dynamic rangeimage. This generation unit comprises an imaging unit 110 which isconfigured to take images or a sequence of n images of an object withdifferent exposures. Since it is necessary that the object, the imagingunit and the surrounding remains static with respect to each other fortaking this first initializing sequence, the imaging unit or thegeneration unit may be arranged on a tripod not shown in FIG. 9.

The generation unit may also comprise a first processor 120 configuredto calculate for each pixel value of the reference image arrayscontaining n−1 difference values, each array corresponding to one of then−1 further images and a second processor 130 configured to calculate amean difference curve for each of the n−1 further images, wherein eachmean difference curve represents a mean difference value for each pixelvalue. The first and the second processor may also be combined as oneprocessor. Further the generation unit 100 may comprise a storage 140which is configured to store the mean difference values, which arerepresented for each pixel value and for each of the n−1 further imagesin a look-up table.

The storage 140 may be configured to store look-up tables for differentfocal lengths and/or different ISO sensitivities.

The generation unit 100 may further comprise an aligning unit 200configured to align the images based on the look-up table and a blendingunit 400 configured to blend the images.

The generation unit 100 may also comprise a linearizing unit 300configured to linearize the images after their alignment.

The generation unit 100 may further comprise an image creating unit 500configured to create a plurality of further images from a single imagebased on the look-up table, which can also be blended by the blendingunit 400.

Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat a variety of alternate and/or equivalent implementations may besubstituted for the specific embodiments shown and described withoutdeparting from the scope of the described embodiments. This applicationis intended to cover any adaptations or variations of the specificembodiments discussed herein. Therefore, it is intended that thisinvention be limited only by the claims and the equivalents thereof.

1. Method for generating a high dynamic range image, comprising: takingwith a camera a first sequence of n images of an object with differentexposures wherein the object and the camera remain static with respectto each other, wherein the sequence contains one reference image withpixel values and n−1 further images with n≧2, calculating for each pixelvalue of the reference image an array containing x difference values,each array corresponding to one of the n−1 further images, wherein x isthe number of times the considered pixel value appears in the referenceimage calculating a mean difference curve for each of the n−1 furtherimages, each mean difference curve representing a mean difference valuefor each pixel value and storing the mean difference values for eachpixel value and for each of the n−1 further images in a look-up table.2. Method for generating a high dynamic range image according to claim 1further comprising taking a second sequence of images with the camera,aligning the images based on the look-up table and blending thelinearized images.
 3. Method for generating a high dynamic range imageaccording to claim 1 further comprising taking a single image, creatinga plurality of further images from the single image based on the look-uptable and blending the single image and the further images to one image.4. Method for generating a high dynamic range image according to claim1, further comprising taking another second sequence of images with thecamera wherein the object and the camera remain static with respect toeach other and blending the images based on the look-up table.
 5. Methodfor generating a high dynamic range image according to claim 1 whereinthe difference between different exposures derives from a difference inexposure times.
 6. Method for generating a high dynamic range imageaccording to claim 1, wherein the difference between different exposuresderives from a difference in aperture size.
 7. Method for generating ahigh dynamic range video comprising taking a sequence of video frames,creating a plurality of further video frames from each video frameaccording to the method of claim 3, wherein each video frame isprocessed as a single image.
 8. Generation unit (100) for generating ahigh dynamic range image, comprising: an imaging unit (110) configuredto take a sequence of n images of an object with different exposureswherein the object and the imaging unit (110) remain static with respectto each other and wherein the sequence contains one reference image withpixel values and n−1 further images with n≧2, a first processor (120)configured to calculate for each pixel value of the reference imagearrays containing n−1 difference values, each array corresponding to oneof the n−1 further images, a second processor (130) configured tocalculate a mean difference curve for each of the n−1 further images,each mean difference curve representing a mean difference value for eachpixel value and a storage (140) configured to store the mean differencevalues for each pixel value and for each of the n−1 further images in alook-up table.
 9. Generation unit (100) for generating a mean differencecurve according to claim 8 characterized in that the storage (140) isconfigured to store look-up tables for different focal lengths and/ordifferent ISO sensitivities.
 10. Generation unit for creating a highdynamic range image according to claim 8, further comprising an aligningunit (200) configured to align the images based on the look-up table anda blending unit (400) configured to blend the linearized images. 11.Generation unit for generating a high dynamic range image according toclaim 8 further comprising an image creating unit (500) configured tocreate a plurality of further images from a single image based on thelook-up table and a blending unit configured to blend the single imageand the further images to one image.
 12. A computer readable mediumincluding computer program instructions that cause a computer to executea method for generating a high dynamic range image, comprising takingwith a camera a sequence of n images of an object with differentexposures wherein the object and a camera remain static with respect toeach other, wherein the sequence contains one reference image with pixelvalues and n−1 further images with calculating for each pixel value ofthe reference image arrays containing n−1 difference values, each arraycorresponding to one of the n−1 further images, calculating a meandifference curve for each of the n−1 further images, each meandifference curve representing a mean difference value for each pixelvalue and storing the mean difference values for each pixel value andfor each of the n−1 further images in a look-up table.
 13. Computerreadable storage medium, comprising a computer readable medium accordingto claim 12.